On the probabilistic properties of the solutions of some high-dimensional optimization problems arising in Statistics

We study the probabilistic properties of the solutions of certain high-dimensional optimization problems arising in statistics. More specifically, if for 1 ≤ i ≤ n, Xi ∈ R and i ∈ R, we study the properties of β̂ = argminβ∈Rp 1 n n ∑ i=1 ρ( i −X ′ iβ) + τ 2 ‖β‖ , in the high-dimensional setting where p/n tends to a finite non-zero limit. While most the work is done for τ > 0, we show that under some extra assumptions on ρ, it is possible to recover the case τ = 0 as a limiting case when p/n < 1. This implies that we can derive results for the unpenalized (i.e τ = 0) standard “regression M-estimate” problem where i is replaced by Yi = X ′ iβ0 + i, with β0 an arbitrary deterministic vector in R, characterizing in this case the behavior of β̂ − β0. Our assumptions on Xi’s are very general and cover for instance cases where Xi’s are i.i.d with independent entries. Importantly, our proof handles the case where these entries are not Gaussian. While our main focus is on the case of i.i.d i’s, our proof technique can also handle the case of i’s with different distributions and we give some details on this problem at the end of the paper.